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ABSTRACT 



This paper presents principles that should be considered in 
planning and interpreting studies of outcomes of interventions with children 
receiving mental health services. Principles to consider when planning 
outcome studies include recognizing that "outcome" refers to multiple rather 
.than single domains and choosing outcome domains that coincide with program 
theory. Principles to consider when interpreting results of outcomes studies 
include the following: (1) consider the source of information when evaluating 

outcome results; (2) recognize that clinical outcomes are not solely the 
result of services; (3) evaluate for adequate statistical power and 
appropriate procedures in data analysis; and (4) remember that conclusions 
are limited by the chosen outcome domains. (DB) 
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Introduction 

Outcomes data have increasingly become a meter by which 
mental health services are evaluated. This is particularly the 
case in this age of managed care, and calls for accountability 
in service delivery where providers are asked to justify 
services on the basis of client outcomes. Outcomes data also 
are crucial for quality assurance and continuous quality 
improvement efforts, as embodied in concepts such as 
outcomes monitoring and outcomes management. However, 
while the movement toward outcome monitoring should be 
applauded, evaluation efforts involving outcomes should not 
be naively undertaken. Service providers, administrators, 
and evaluators should seriously consider a number of 
important theoretical and research issues that can limit 
conclusions made from outcomes data and research studies. 
A number of principles should be considered when planning 
outcome studies and especially when interpreting research 
results. 
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Principles to Consider when Planning Outcomes 

Recognize that "outcome" refers to multiple rather than 
a singular domain. 

The word "outcome" has been used many times without 
outlining what is actually meant. A vague use of the term 
seems common. Multiple outcomes are generally 
acknowledged, but the apparent lack of specificity when 
discussing outcomes suggests a need for further 
conceptual izati on . 

Hoagwood, Jensen, Petti, & Bums (in press) developed an 
excellent conceptual model of outcomes that attends to 
multiple levels (e.g., individual, family, and community) 
and contexts (e.g., home, school, and peer group). Their 
model consists of five domains: (a) symptomatic/diagnostic 
(e.g., depression, aggression, and diagnosis); (b) functional 
(e.g., child's adaptation to various environments); (c) 
consumer perspectives (e.g., client satisfaction and family's 
quality of life); (d) environmental (e.g., changes in marital 
relationship or school environment); and (e) services and 
systemic (e.g., reductions in out-of-home placements, 
hospitalization rates, and length of stay). 

I would add one additional outcome 
domain&endash;protection of child and family rights and 
dignity. This domain would include no harm occurring to 
the child or family during treatment, maintenance of 
confidentiality, provision of effective services, and 
outcomes associated with consumer empowerment, such as 
opportunities for family involvement in developing agency 
policies and services as well as family involvement in 
treatment, especially a significant "voting" role in treatment 
decisions. 

Choose outcome domains that coincide with the 
program theory. 

It is important to select outcome domains that match the 
service program being evaluated. What outcomes would you 
reasonably expect to be associated with an adequate 
implementation of your program theory? What additional 
outcomes would your program like to meet? 

This principle became particularly clear as a result of some 
work recently done by the author to define quality indicators 
with an agency that provides services to children and 
adolescents who have had significant contacts with the 
juvenile justice system. They were asked to list crucial 
elements associated with producing positive clinical 
outcomes. The first question asked was what was meant by 
clinical outcomes, as they recently had problems developing 
a shared understanding of this term in a meeting with a 
managed care company. This agency recognized that they 
might not be able to significantly impact clinical symptoms 
in their programs, but hoped to have an impact on 
functioning, especially societal, school, and vocational 




3 



functioning, especially societal, school, and vocational 
functioning. They also were very concerned with creating a 
safe environment (i.e., protecting client rights) for those in 
their program. 

Principles to Consider when Interpreting 
Outcome Results 

The source of information should be considered when 
reflecting on outcome results. 

Constructionist theories suggest that each individual has 
their own unique perspective on the world. Truth is truly in 
the eye of the beholder. In the case of collecting outcomes 
data, it seems important to consider how different sources 
(e.g., parents, clinicians, children, and teachers) may 
contribute different information. Many possibilities exist 
that might explain why we would obtain different outcome 
results from different sources, including: (a) each source has 
a different sample of a particular child's behavior from 
which to make outcome judgments, and (b) points of 
reference for making judgments differ. 

In regards to the first reason, parents may be good reporters 
of some behaviors, but may be less aware of certain aspects 
of their child's life. For example, the amount of information 
a parent has about a child's school and social life may vary 
considerably. The level of parent involvement would affect 
the validity of parent reports in these areas. Another area of 
interest concerns the validity of reports of externalizing and 
internalizing behaviors. Parents are generally thought to be 
better reporters of externalizing behaviors, because they are 
more likely to come to their attention. Adolescents are 
viewed as better reporters of internalizing behaviors. 
Teachers may provide valid reports of academic and social 
behavior, but fewer valid reports for other behaviors. 

Service providers likely have very limited knowledge of the 
child's behavior outside of the service setting. 

In addition to interacting with the child in different contexts, 
different sources of information likely have different points 
of reference from which to make their judgments. In other 
words, the sample to which the child is being compared for 
making judgments is likely to differ. Response categories on 
various scales (e.g., somewhat or sometimes true, a lot or a 
little, more than average or less than average) may have 
different meanings to different judges. For instance, teachers 
are very familiar with a wide-range of children and may 
have a different sense of what "average" or "a lot" mean, 
compared to parents and clinicians. 

Recognize that clinical outcomes are not solely the result 
of services. 

Multiple factors impact the mental health of children. These 
include family influences, peer influences, cultural 
influences, economic influences, and strengths and 
resiliency. These factors must be considered as additional 
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resiliency. These factors must be considered as additional 
influences on changes in symptoms and functioning that 
occur between intake and exit from services. In a review of 
research on the impact of disasters on the psychological 
health of adults and children (Salzer & Bickman, in press), 
one consistent finding is that disasters are associated with 
high levels of psychological distress immediately following 
the traumatic event. However, as the community and 
families recover, usually without the aid of mental health 
interventions, psychological problems remit. One 
explanation for these results is based on a central tenet of 
ecological theories&endash;natural systems work to return 
to stasis following a disruption. 

Children and families entering services do so at a time of 
great upheaval and are generally found to be improved at 
the end of treatment. However, the role of services in 
bringing about such changes often fails to be separated from 
non-treatment effects. The natural resiliency and resources 
of children and family systems likely contribute to 
improvements over time. Children and families, either 
consciously or unconsciously, work to make changes that 
alleviate stressors, with and without help from service 
providers. Children, parents, and other family members also 
rely on their natural resources, such as friends and other 
family members, in addition to the support and guidance 
provided by the service system. The fact that even 
non-treatment control groups may improve over time 
supports the notion that natural recovery processes are 
important factors to consider when making causal 
statements about the relationship between treatment and 
improvement. 

Use of control groups would be one obvious approach to 
take in order to distinguish between natural recovery 
processes and treatment effects. The development of 
long-term norms on various measures for those with severe 
problems who have not received services might be another 
approach. The clinical outcomes for one child, gathered at 
6-months post-intake, could be compared to the 6-month 
norms for children who also scored high on the same 
instrument at time 1 but received no treatment. 

Adequate statistical power and appropriate procedures 
are necessary when analyzing data. 

Statistical conclusion validity (Bickman, 1990) is a 
significant problem with current research on outcomes 
associated with measures of mental health. Statistical 
conclusion validity refers to the researchers' ability to make 
valid statistical conclusions, including the ability to detect 
an effect when one is present (i.e., power) and when one is 
not present (i.e.. Beta or type II error). One concern is that 
many studies lack sufficient statistical power to detect small 
or medium effect sizes, due to small samples. This may, in 
fact, account for some of the null results and inconsistent 
results reported in the literature for various interventions. 
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While it depends on the effect size that one expects to be 
associated with the intervention, in general, researchers 
should strive for large sample sizes whenever possible. 

While lack of statistical power may limit the number of 
studies finding positive effects for community interventions, 
a great number of studies also include multiple, unplanned 
statistical analyses that inflate the familywise error rate to 
such an extent that erroneous conclusions might be made 
suggesting an intervention is effective when it may not be 
(i.e., alpha or type I error). For example, it appears common 
for researchers to conduct multiple univariate tests without 
the proper corrections for inflated alpha levels. 

Conclusions are limited by the outcome domains you 
choose. 

Results from one outcome domain cannot be used as a 
proxy for other domains. This is a particularly important 
concept to keep in mind when attempting to assess the 
clinical effectiveness of services, without obtaining data on 
clinical symptoms or functioning. For example, reductions 
in service utilization, such as the number of out-of-home 
placements or decreased rates of hospitalization, may be 
reported to imply that a service system or a particular 
service like case management are clinically effective. 
However, service utilization outcomes may predominantly 
reflect modifications in system priorities and policies 
regarding level of care or length of stay decisions rather 
than actual changes in the child's behavior (e.g., one of the 
philosophies associated with a system of care places an 
emphasis on keeping children in their homes, or at least 
their home communities, and this philosophical change 
alone may account for decreases in out-of-home placements 
and hospitalization rates regardless of changes in the child's 
behavior). While extremely important, service utilization 
outcomes are of little benefit in evaluating the clinical 
effectiveness of a system of care. 

There appears to be another current trend toward reporting 
services to be effective based on results from consumer 
satisfaction instruments. Consumer satisfaction is vitally 
important to how services should be evaluated, but the 
relationship between satisfaction and clinical outcomes is 
unclear (Lambert, Salzer, Bickman, 1996). The best way to 
assess clinical outcomes may be through the use of 
measures assessing symptoms and functioning. 
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